Vehicle information photo overlay

ABSTRACT

An image information overlay system retrieves an image associated with a vehicle listing and uses machine learning models to classify the image, generating identification data that may comprise a vehicle make and model, a feature or part of the vehicle present in the image, and a location of the vehicle feature or part. The identification data or an individual identifier of the vehicle, such as a Vehicle Identification Number (VIN), may be used to retrieve overlay information related to the vehicle make and model, such as recalls or known maintenance issues or information specific to the vehicle, such as mileage, accident reports, or ownership history. The overlay information is displayed on the image as an overlay at the location of the vehicle feature or part corresponding to the overlay information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/742,210 filed on Jan. 14, 2020, which is incorporated by referenceherein.

BACKGROUND

A listing for a new or used vehicle for sale may be displayed topotential buyers through a web interface. The listing often includes atext portion and one or more images of the vehicle offered for sale.Vehicle features and specifications may be displayed in the text portionof the web interface. Vehicle title history, such as mileage, accidentreports, recalls, etc. corresponding to the vehicle may be availablethrough a third-party vehicle title history service. Potential buyersbear the burden of finding information they consider relevant to theirpurchase decision, such as vehicle features, specifications, and titlehistory, wherever that information is displayed in the text portion ofthe web interface or by accessing another third-party service.

BRIEF SUMMARY

In some embodiments, a method of overlaying information on an imagecomprises retrieving an image using an image identifier and classifyingthe image to generate identification data and a feature location for theimage. Based on the identification data and/or an individual identifiercorresponding to the image, the method retrieves overlay informationfrom a database. The method may determine a location of the image in adisplay and may display the overlay information at the feature locationin the image.

In some embodiments, a method for overlaying information on an imagecomprises selecting an image identifier in a hypertext document andclassifying an image identified by the image identifier to generateidentification data for the image. The method identifies a featurelocation within the image. The method may select overlay informationbased on the identification data and display the overlay information onthe image based on the feature location.

In some embodiments, the method further comprises identifying a featureat the feature location and selecting the overlay information based on afeature identifier corresponding to the identified feature.

In some embodiments, the method further comprises identifying anindividual identifier in the hypertext document and selecting theoverlay information based on the individual identifier.

In some embodiments, the selecting of the image identifier comprisesselecting, as the image identifier, an image identifier nearest, in adisplay of the hypertext document, to the identified individualidentifier.

In some embodiments, the method further comprises selecting the imageidentifier based on one or more of a threshold of an image sizecorresponding to the image identifier, a ratio between a length of theimage and a width of the image, or visibility, in a display of thehypertext document, of an image corresponding to the image identifier.

In some embodiments, the displaying of the overlay image comprisesdetermining, based on a layout of a display of the hypertext document, alocation of the image identified by the image identifier and displayingthe overlay information at the feature location in the image based onthe location of the image in the display of the hypertext document.

In some embodiments, the method uses a first machine learning model toclassify the image in a first general image class and uses a secondmachine learning model to generate a general identifier for the image.The general identifier may correspond to one of multiple sub-classeswithin the first general image class. The selecting of the overlayinformation may comprise selecting the overlay information based on thegeneral identifier.

In some embodiments, the method further comprises using a third machinelearning model to identify a feature in the image and determine thefeature location in the image. The third machine learning model maygenerate identification data comprising a feature identifiercorresponding to the identified feature. The third machine learningmodel may be configured to identify the feature based on an image setcorresponding to the general identifier.

In some embodiments, the method selects multiple image identifiers andclassifies the images identified by the multiple image identifiers todetermine an image in a first general image class and generate thegeneral identifier for the determined image.

In some embodiments, systems and computer program products of thedisclosed embodiments may include a computer-readable device storingcomputer instructions for any of the methods disclosed herein or one ormore processors configured to read instructions from the computerreadable device to perform any of the methods disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

The accompanying drawings are incorporated herein and form a part of thespecification.

FIG. 1 is a block diagram of an image information overlay systemaccording to some embodiments.

FIG. 2 is a flow chart for a method of image information overlayaccording to some embodiments.

FIG. 3 is block diagram of an image classifier used for imageinformation overlay according to some embodiments.

FIG. 4 is a flow chart for a method of image information overlay,performed by a client device, according to some embodiments.

FIG. 5 is a flow chart for a method of image information overlay stepsperformed by a server according to some embodiments.

FIG. 6 is a flow chart for a method of image information overlay formultiple images according to some embodiments.

FIG. 7 is a non-limiting example diagram of a web browser interface forimage information overlay according to some embodiments.

FIG. 8 is a block diagram of an example computer system useful forimplementing various embodiments disclosed herein.

In the drawings, like reference numbers generally indicate identical orsimilar elements. Additionally, generally, the left-most digit(s) of areference number identifies the drawing in which the reference numberfirst appears.

DETAILED DESCRIPTION OF THE INVENTION

The embodiments described with reference to the figures describe asystem and method for displaying vehicle information overlaid on avehicle image. Vehicle information may be overlaid on vehicle featuresor parts corresponding to the overlay information. The embodiments aredescribed with specific examples given in relation to a web pluginconfigured to display the overlay information on images displayed invehicle listings in a web interface. However, a person of ordinary skillin the art would understand that the principles disclosed herein may beapplied in other user interfaces, including a vehicle dealershipdatabase, vehicle configurators, vehicles listed by individual sellerson a computerized marketplace, or similar applications using any kind ofcomputing device.

FIG. 1 is a block diagram of an image information overlay systemaccording to some embodiments.

According to some embodiments, the image information overlay systemcomprises a client 101 computer system and a server 104. The client 101computer system and the server 104 may be configured to perform thefunctions of the image information overlay system as described herein.According to some embodiments, the client 101 may comprise a web browser102, and a plugin 103 configured to augment the browser with one or moreaspects of the image information overlay system functionality. Accordingto some embodiments, the server may be configured to host an imageclassifier 105. The image classifier 105 may comprise one or moremachine learning (ML) models configured to classify a vehicle image andgenerate identification data for the image. Identification data maycomprise one or more of a feature identifier that describes a feature ofthe image, a feature location indicating the location of the feature inthe image, a general image class (e.g., vehicle, non-vehicle, vehicleexterior, vehicle interior), a general identifier corresponding to thevehicle in the image, and/or an individual identifier for the vehicle inthe image. One or more of the machine learning models of the imageclassifier 105 may be configured to identify a feature in the image anda feature location in the image. According to some embodiments, theoverlay information may be displayed at the feature location on theimage.

Some machine learning models applicable to the disclosed embodiments maybe configured to perform image recognition and/or image classificationtasks. Image classification may include classifying an image as avehicle image, a vehicle exterior, a vehicle interior, a three-quartersprofile vehicle image or another vehicle perspective image. For some ofthe specific vehicle features, detecting the location of the feature inan image and returning box coordinates may be preferred. Machinelearning models and neural network architectures of the disclosedembodiments may include, but are not limited to, Convolutional NeuralNetworks (CNNs), Inception, ResNet and NASNet, Region-based CNN (R-CNN),You Only Look Once (YOLO), and Single Shot Detector (SSD). The disclosedembodiments may also employ other machine learning approaches includinglogistic regression, support vector machines (SVM), or tree-basedmethods like random forests and gradient boosting machines (GBM).Preprocessing techniques, also known as feature creation or featuretransformation, may also be used to convert the pixel data of the imageinto features for the machine learning models to identify. Somepreprocessing techniques may include, but are not limited to, histogramof oriented gradients (HOG), scale-invariant feature transform (SIFT),or features from accelerated segment test (FAST). A non-neuralnetwork-based object detection method of the disclosed embodiments mayinclude a Viola-Jones object detection framework, which uses Haarfeatures.

Convolutional Neural Networks (CNNs) are current state-of-the-artmachine learning modeling techniques for image classification problems,with other useful neural network architectures including Inception,ResNet and NASNet for image recognition, and Region-based CNN (R-CNN),You Only Look Once (YOLO), and Single Shot Detector (SSD) for objectdetection. It is also possible to apply other machine learningapproaches including logistic regression, support vector machines (SVM),or tree-based methods like random forests and gradient boosting machines(GBM). These other machine learning approaches may involve usingpreprocessing techniques, such as feature creation or featuretransformation, to convert the pixel data of the image into features theother machine learning approaches can operate on. These preprocessingtechniques may include histogram of oriented gradients (HOG),scale-invariant feature transform (SIFT), or features from acceleratedsegment test (FAST).

According to some embodiments, the client 101 and server 104 areconnected by a network 110. The plugin 103 may be configured to interactwith the image classifier 105 by connecting to the server through thenetwork 110. According to some embodiments, the plugin 103 may beconfigured to identify an image in a hypertext document displayed in theweb browser 102, and send a request for image classification to theimage classifier 105. The hypertext document may be a vehicle listing.The functions of the client and server may also be implemented on asingle computer system, in a cloud computing environment, or through anycomputing system configuration.

According to some embodiments, the image information overlay system maydisplay overlay information on a vehicle image. The overlay informationmay comprise information specific to the individual vehicle, such asmileage, accident reports, maintenance history, or ownership history orinformation generally applicable to the make and model of the vehicle,such as known maintenance issues, manufacturer recalls, or real-worldfuel economy data. Overlay information may be retrieved based onidentification data available in a hypertext document associated withthe vehicle, or based on detecting a vehicle make and model using animage classifier 105. The identification data available in the hypertextdocument may comprise an individual identifier, such as a VehicleIdentification Number (VIN) associated with the vehicle, and/or ageneral identifier such as an indicator of the vehicle make and model.The image classifier 105 may also generate identification data used toretrieve overlay information.

Overlay information may be overlaid on the image at a feature location,in the image, corresponding to the overlay information being displayed.An image classifier 105 may be used to identify the feature location, inthe image, where the overlay information is to be displayed. Accordingto some embodiments, the image information overlay system may displayoverlay information in a general location of the image, not specific toany particular feature of the image, or in a location that does notobscure the vehicle in the image. Overlay information may be displayeddirectly on the image or in an overlay information pop-up. An overlayinformation pop-up may be associated with an icon displayed in a generallocation on the image, or displayed at a feature location in the image.

FIG. 2 is a flow chart for a method of image information overlayaccording to some embodiments. It is to be appreciated the steps may notoccur in the order shown, not all steps may be performed, and additionalsteps may be performed, depending on various applications.

According to some embodiments, the image information overlay system maybe configured to operate on a hypertext document displayed in a webbrowser. The image information overlay system may be configured as a webplugin which operates automatically on a whitelist of domains. Thewhitelist of domains may comprise a list of domains known to be avehicle dealership site, a vehicle classified advertisement site, avehicle review site, or another site having vehicle images and vehicleinformation. When the browser accesses a domain on the whitelist, theimage information overlay system may be triggered to search for vehicleimages. At step 201, the image information overlay system selects animage identifier in the hypertext document. The image identifier maycomprise a path to a local or remote storage location. The path maycomprise a universal resource locator (URL) identifying a location wherethe image is stored. The image information overlay system may select theimage identifier based on one or more of the position of the image inthe hypertext document, image size, a ratio of image length and width,or visibility of the image.

According to some embodiments, the image information overlay system mayidentify an individual identifier in the hypertext document and selectthe image identifier based on proximity to the individual identifier.The image information overlay system may select the image identifiernearest to the individual identifier in the display of the hypertextdocument in the web browser 103. The individual identifier may be avehicle identification number (VIN) included in the hypertext document.

At step 202, the image information overlay system may retrieve the imageidentified by the image identifier. The image may be retrieved byaccessing the network location identified by the URL and copying theimage to local storage. In the case that the path identifies a localstorage location, the image may be retrieved by copying the image intomemory suitable for the image classifier 105 to perform imageclassification on the image.

At step 203, the image information overlay system uses the imageclassifier 105 to classify the image and generate identification datafor the image corresponding to the image identifier. According to someembodiments, the identification data may comprise a general identifier.The general identifier may indicate a make and model of the vehicle. Atstep 204, the image information overlay system may use the imageclassifier 105 to identify a feature location corresponding to one ormore features identified in the image.

At step 205, the image information overlay system retrieves overlayinformation based on the identification data. According to someembodiments, the image information overlay system accesses one or morevehicle databases to retrieve overlay information, such as knownmaintenance issues, manufacturer recalls, or real-world fuel economydata. The overlay information may correspond to a general identifierindicating a make and model of the vehicle in the identification data.The one or more vehicle databases may comprise a third-party databaseand/or a local database storing information generally applicable tovehicles based on the make and model of the vehicle.

According to some embodiments, the image information overlay system mayaccess one or more vehicle databases and use an individual identifier,from the identification data, to retrieve overlay information specificto the vehicle, such as mileage, accident reports, maintenance history,or ownership history. The individual identifier may be a vehicleidentification number (VIN) included in the hypertext document. The oneor more vehicle databases may comprise a vehicle title history database.

According to some embodiments, the image information overlay systemretrieves overlay information based on identified features included inthe identification data. As a non-limiting example, the identificationdata may comprise a feature of alloy wheels. The image informationoverlay system may access one or more vehicle databases to retrieveoverlay information identifying whether the alloy wheels are stockwheels or a dealer installed option, or installed by a previous orcurrent owner.

Overlay information may be mapped to particular features andcorresponding feature locations in the identification data. For example,a known maintenance issue, retrieved based on the general identifier andrelated to the brakes of a particular make and model, may be mapped tothe identified feature of the wheels in the identification data. Asanother non-limiting example, one or more of mileage, known maintenanceissues, engine specifications, or other mechanical information may bemapped to a feature of a hood of the vehicle. At step 206, the imageinformation overlay system may display the retrieved overlay informationon the image based on the feature location. According to someembodiments, the image information overlay system may position theoverlay information at or near the feature location identified in theimage, such that some portion of the overlay information overlaps withthe feature location. According to some embodiments, overlay informationmay be displayed at any location on the image. As a non-limitingexample, overlay information that is general to the vehicle, and notrelated to any specific feature identified in the image, may bedisplayed at any location in the image, displayed in a pop-up near theimage, or displayed in a location of the image that does not obscure thevehicle in the image.

FIG. 3 is a block diagram of an image classifier 105 used for imageinformation overlay according to some embodiments.

According to some embodiments, the tasks of classifying a vehicle image,identifying a feature location in the vehicle image, and generatingidentification data may be divided into multiple sub-tasks anddistributed among multiple machine learning models. Distributing imageclassification into separate machine learning models may narrow therange of images that each machine learning model is configured toclassify, and may increase the operation speed, improve classificationaccuracy, and reduce complexity of the machine learning models. Forexample, vehicle interior images may have much more variability andcomplexity than vehicle exterior images, while sharing few visualsimilarities. Using a single machine learning model to classify allimages in the full range of possible vehicle exterior and interiorimages can increase difficulty of training the machine learning modeland reduce the total accuracy and speed of the machine learning model.However, a machine learning model that can focus on determining if animage comprises a vehicle exterior, can filter the inputs to anothermachine learning model that can focus on determining if the imagecomprises a vehicle interior, simplifying the job of both machinelearning models.

According to some embodiments, one or more machine learning models maybe configured to generate identification data for an image as a resultof classifying the image. Identification data may comprise an indicatorthat the image contains a vehicle, a general identifier indicating amake or make and model of the vehicle, or any feature of the vehicle.Features of the vehicle image that may be included in the identificationdata include, but are not limited to, one or more of a viewingperspective of the vehicle in the image, a body style, trim level, bodykit, special paint job, accessory rack, front or rear bumpers, hood,number of doors, rear door, spare tire, wheels, wheel size and/or spokenumber, alloy wheels, exhaust accessories, spoilers, scoops, a bodypart, a mechanical part, automatic/manual transmission, seat style,seating arrangement, seat material, climate control features,navigation, rear headrest screens, split rear folding seats, performancefeatures, dashboard features, entertainment features, safety features, asunroof or any other exterior or interior feature of a vehicle. Afeature of the vehicle identified in the vehicle image may berepresented in the identification data by a feature identifier.According to some embodiments, one or more of the machine learningmodels are configured to identify a feature location in the image for afeature identified in the image. According to some embodiments, thefeature location comprises a point, defined by x and y coordinatevalues, identifying a location in the image where the feature wasidentified. According to some embodiments, the feature locationcomprises three or more points which, when connected by lines, provide ageometric shape that overlaps the feature identified in the image.

According to some embodiments illustrated in the diagram of FIG. 3, atstep 301, the image classifier 105 uses a first machine learning model,machine learning model 1, to classify an image and generateidentification data comprising a first general image class. The firstgeneral image class may comprise vehicle exterior images. According tosome embodiments, the image may be classified as a vehicle if the imagecomprises a vehicle exterior, and the image may be unclassified if theimage does not contain a vehicle exterior.

According to some embodiments, the image classifier 105 sends imagesclassified as vehicle exterior images to a second machine learningmodel, machine learning model 2 302, for further classification. Machinelearning model 2 302 may classify the image and generate identificationdata comprising a general identifier corresponding to one or more ofmultiple sub-classes within the first general image class. The generalidentifier may comprise vehicle make and model within the class ofvehicle exterior images. Because the image sent to machine learningmodel 3 has already been classified as an image comprising a vehicleexterior, the classification of vehicle make and model is simplified andaccuracy may be improved.

According to some embodiments, an image classified as comprising aparticular make and model of vehicle may be sent to a third machinelearning model, machine learning model 3 303, for further classificationto generate identification data comprising a feature of the image.Machine learning model 3 may be configured to classify an image based ona vehicle perspective, such as a side perspective, front perspective,three-quarters perspective, rear perspective or any other perspective.

According to some embodiments, an image classified by machine learningmodel 3 may be sent for further classification to one or more of machinelearning model 4 a 304, machine learning model 4 b 305, or machinelearning model 4 c 306, depending on which vehicle perspective the imagewas classified in by machine learning model 3. Each of machine learningmodels 4 a, 4 b, 4 c, and 4 d may be configured to identify features ofthe vehicle from a different perspective of the vehicle.

Different vehicle perspectives may expose different features of thevehicle in the image. For example, the wheels may be more visible in aside or three-quarters perspective than in a front perspective, whilethe hood and windshield may be more visible from a front perspective,and the rear bumper may be entirely hidden from the front perspective.Dividing the machine learning models into different perspectives canreduce the total number of distinct features each machine learning modelis configured to identify.

According to some embodiments, machine learning models 4 a, 4 b, and 4c, are configured to identify features of the vehicle in the imagecomprising one or more of a body style, trim level, body kit, spoilers,scoops, special paint job, accessory rack, front or rear bumpers, hood,number of doors, rear door, spare tire, wheel size and/or spoke number,alloy wheels, exhaust accessories, or any other external features of avehicle. A particular machine learning model may be configured toidentify a different set of features depending on a make and model ofthe vehicle.

The order in which machine learning model 2 302 and machine learningmodel 3 303 classify the images need not be in the order illustrated inFIG. 3 and some machine learning models may be optional. For example, insome embodiments, machine learning model 3 303 may be eliminated and oneor more of machine learning models 4 a 304, 4 b 305, or 4 c 306 may beconfigured to further classify the image to detect individual featuresof the vehicle. Therefore the third machine learning model may refer toone or more of machine learning model 3 303, machine learning model 4 a304, machine learning model 4 b 305, or machine learning model 4 c 306.Machine learning models 4 a 304, 4 b 305, or 4 c 306 may be configuredto further classify the image in series or in parallel. According tosome embodiments, the third machine learning model may comprise a singlemachine learning model configured to identify vehicle exterior featuresfrom any perspective.

According to some embodiments, the image classifier 105 may beconfigured to use machine learning models 4 a 304, 4 b 305, or 4 c 306to detect features common to all makes and models before sending theimage to machine learning model 3 303 for classification by make andmodel.

According to some embodiments, the image classifier 105 may beconfigured to send images not classified in the first general imageclass to a second machine learning model, machine learning model A 310,for classification in a second general image class. The second generalimage class may comprise vehicle interior images. The image classifier105 may be configured to treat images that are not classified as vehicleexterior images or vehicle interior images as unclassified images orimages that contain no vehicle.

According to some embodiments, the image classifier 105 may beconfigured to send images classified as vehicle interior images to athird machine learning model comprising one or more of machine learningmodel B1 311, machine learning model B2 312, machine learning model B3313, or machine learning model B4 314. Each of machine learning modelsB1, B2, B3 and B4 may be configured to detect a different set of one ormore features of the vehicle interior. According to some embodiments,machine learning models B1, B2, B3, and B4 are configured to detectinterior features including, but not limited to, automatic/manualtransmission, seat style, seating arrangement, seat material, climatecontrol features, navigation, rear headrest screens, split rear foldingseats, performance features, dashboard features, entertainment features,safety features, a sunroof, or any other interior feature of a vehicle.

FIG. 4 is a flow chart for a method of image information overlay,performed by a client 101 device, according to some embodiments. It isto be appreciated the steps may not occur in the order shown, not allsteps may be performed, and additional steps may be performed, dependingon various applications. According to some embodiments, a client 101device includes a browser 102, and a plugin 103 augments the browser 102functionality with one or more aspects of the image information overlaysystem.

At Step 401, the image information overlay system identifies anindividual identifier in a hypertext document displayed in the webbrowser. The individual identifier may be a VIN, and the plugin 103 maybe configured to recognize the VIN based on a distinct format including17 characters, digits and letters, that act as a unique identifier forthe vehicle.

At step 402, the image information overlay system selects an imageidentifier from the hypertext document. The image identifier may be auniversal resource locator identifying a location where the image isstored. The plugin 103 may select the image identifier based onproximity to the individual identifier. The plugin 103 may select theimage identifier nearest to the individual identifier in the display ofthe hypertext document in the web browser 103. The plugin 103 may selectthe image identifier based on one or more of the position of the imagein the hypertext document, image size, a ratio of image length andwidth, or visibility of the image.

According to some embodiments, at step 403, the plugin 103 of the imageinformation overlay system sends the image identifier to a server 104hosting an image classifier 105. The plugin 103 may send the selectedimage identifier to the server 104 of the image information overlaysystem with a request for identification data and/or a feature location.At step 404, the plugin 103 receives, from the server 104,identification data and a feature location.

At step 405, the image information overlay system retrieves overlayinformation corresponding to the identification data and/or theindividual identifier. The plugin 103 may access one or more vehicledatabases to retrieve overlay information. The one or more vehicledatabases may comprise a third-party database and/or a local databasestoring information generally applicable to vehicles based on the makeand model of the vehicle, or a vehicle title database storing vehicleinformation based on VINs. The overlay information may comprise knownmaintenance issues, manufacturer recalls, or real-world fuel economydata corresponding to the make and model in the identification data, oroverlay information specific to the vehicle, such as mileage, accidentreports, maintenance history, or ownership history.

At step 406, the image information overlay system determines a locationof the image in a display of the hypertext document. The plugin 103 maydetermine the location of the image based on a layout of a display ofthe hypertext document.

At step 407, image information overlay system may display the overlayinformation on the image based on the feature location and the locationof the image in the display of the hypertext document. The location ofthe image may be a point comprising an x coordinate and a y coordinatein relation to a reference point of a display area of the browser 102.The feature location may be a point comprising an x coordinate and a ycoordinate in relation to a reference point of the image. The plugin 103may determine the location, in relation to a reference point of adisplay area of the browser, to display the overlay information based ona sum of the feature location in the image and the location of the imagein the display area of the browser.

FIG. 5 is a flow chart for a method of image information overlay stepsperformed by a server according to some embodiments. It is to beappreciated the steps may not occur in the order shown, not all stepsmay be performed, and additional steps may be performed, depending onvarious applications.

According to some embodiments, the image classifier 105 may beimplemented on a server 104, and the server 104 receives an imageidentifier from a client 101 at step 501. The image identifier maycomprise a path to a local or remote storage location. The path maycomprise a universal resource locator identifying a network locationwhere the image is stored.

At step 502, image information overlay system retrieves the imageidentified by the image identifier. The image may be retrieved byaccessing the network location identified by the universal resourcelocator and copying the image to local storage. In the case that thepath identifies a local storage location, the image may be retrieved bycopying the image into memory suitable for the image classifier 105 toperform image classification steps on the image.

At step 503, the image classifier 105 of the image information overlaysystem classifies the image and generates identification data. The imageclassification may be broken down into multiple image classificationsteps using multiple machine learning models configured to classify theimage first in a general image class and then in a more specific imageclass within the general image class. One or more machine learningmodels of the image classifier 105 may be configured to identify afeature and a feature location in the image.

At step 503 a, the image classifier 105 of the image information overlaysystem uses a first machine learning model to classify the image in afirst general image class. The first general image class may comprisevehicle images. The first general image class may also comprise vehicleexterior images or vehicle interior images.

At step 503 b, the image classifier 105 uses a second machine learningmodel to classify the image to determine a general identifier. Thegeneral identifier may be an identifier of one or more sub-classeswithin the first general image class. According to some embodiments, thegeneral identifier comprises a vehicle make or a vehicle make and model.

At step 503 c, the image classifier 105 uses a third machine learningmodel to identify a feature in the image. The third machine learningmodel may generate a feature identifier identifying the feature in theimage. The third machine learning model may identify multiple featuresin the image and a feature identifier for each of the multiple features.The identification data may further comprise the feature identifier.

At step 503 d, the image classifier 105 identifies a feature location inthe image. The identification of the feature location in the image maybe performed by a third machine learning model. The third machinelearning model may perform identification of one or more features in theimage, one or more feature identifiers corresponding to the one or morefeatures, and may also perform identification of one or morecorresponding feature locations corresponding to the identifiedfeatures.

According to some embodiments, the server 104 may retrieve overlayinformation corresponding to the identification data. The server 104 mayaccess one or more vehicle databases to retrieve overlay information.The one or more vehicle databases may comprise a third-party databaseand/or a local database storing information generally applicable tovehicles based on the make and model of the vehicle. The overlayinformation may comprise known maintenance issues, manufacturer recalls,or real-world fuel economy data corresponding to the make and model inthe identification data.

Overlay information may be mapped to particular features andcorresponding feature locations in the identification data. For example,a known maintenance issue related to the brakes of a particular make andmodel may be mapped to the identified feature of the wheels in theidentification data. As another example, one or more of mileage, knownmaintenance issues, engine specifications, or other mechanicalinformation may be mapped to a feature of a hood of the vehicle.

At step 504, the server 104 sends the identification data, featurelocation, and/or overlay information to the client 101. Theidentification data and feature location may be viewed as separate dataor the feature location may be included in the identification datawithout departing from the spirit and scope of the disclosedembodiments.

FIG. 6 is a flow chart for a method of image information overlay formultiple images according to some embodiments. It is to be appreciatedthe steps may not occur in the order shown, not all steps may beperformed, and additional steps may be performed, depending on variousapplications.

The image information overlay system may be configured to perform imageinformation overlay on multiple images in the display of a hypertextdocument. The image classifier 105 may be configured to perform imageclassification on multiple images using multiple image identifiers. Asdescribed in reference to FIG. 2, the image classifier 105 may beconfigured to treat images that are not classified as vehicle exteriorimages or vehicle interior images as unclassified images or images thatcontain no vehicle. For images that contain no vehicle, the imageclassifier may be configured to skip the feature identification machinelearning models and return empty identification data and no featurelocation corresponding to the non-vehicle images. This feature preventsdisplay of overlay information on non-vehicle images that may bedisplayed in the display area of the browser 102.

At step 601, the image information overlay system selects multiple imageidentifiers in the hypertext document. The hypertext document maycomprise one or more images and a vehicle listing with an individualidentifier identifying the vehicle. Some of the one or more images inthe hypertext document may be images of the vehicle corresponding to thevehicle listing and some of the one or more images in the hypertextdocument may be non-vehicle images that are unrelated to the vehicle ofthe vehicle listing.

The image information overlay system may select multiple imageidentifiers from the hypertext document based on attributes of theimages corresponding to the image identifiers. Attributes may compriseproximity of the image to the individual identifier in a display of thehypertext document, position of the image in the hypertext document,image size, a ratio of image length and width, or visibility of theimage in a display of the hypertext document.

According to some embodiments, images unrelated to the vehicle of thevehicle listing may not be selected, based on the attributes of theimages. According to some embodiments, images that are clearly notvehicle images, based on these attributes, are not sent to the imageclassifier 105 for image classification. The selected multiple imageidentifiers may correspond to different images of the vehicle. One ormore of the selected multiple image identifiers may correspond tonon-vehicle images displayed in the hypertext document. According tosome embodiments, the image classifier may be configured to distinguishbetween vehicle images and non-vehicle images and identify vehicleimages among the multiple images corresponding to the selected multipleimage identifiers.

According to some embodiments, at step 602, the image informationoverlay system retrieves the multiple images corresponding to themultiple image identifiers. According to some embodiments, at step 603,the image information overlay system classifies the images to determineone or more images, among the multiple images, in the first generalimage class. According to some embodiments. The first general imageclass may comprise vehicle images. Image classification may be performedaccording to any of the embodiments described herein.

According to some embodiments, at step 604, the image informationoverlay system identifies a feature in the determined one or more imagesthat were classified in the first general image class. The imageinformation overlay system may identify multiple features in each ofmultiple images classified in the first general image class. Accordingto some embodiments, at step 605, the image information overlay systemgenerates identification data and identifies the feature location in thedetermined one or more images. The image information overlay system maygenerate identification data and identify the feature location formultiple features identified in each of multiple images classified asvehicle images. The identification data may include the imageidentifiers for the determined one or more images and one or morefeature identifiers corresponding to one or more identified features inany of the determined one or more images.

According to some embodiments, the image information overlay system maybe configured to filter out images that the image classifier 105 failsto classify as a vehicle image. The image information overlay system mayperform feature identification and generate identification data formultiple images classified as vehicle images, and perform no featureidentification and no identification data for images that the imageclassifier 105 fails to classify as a vehicle image. According to someembodiments, image identifiers of the non-vehicle images are excludedfrom the identification data generated by the image classifier 105, and,therefore, no overlay information is displayed on the non-vehicleimages.

According to some embodiments, at step 606, the image informationoverlay system retrieves overlay information based on the identificationdata of the determined one or more images and/or the individualidentifier. Overlay information may be retrieved according to any of theembodiments disclosed herein. According to some embodiments, at step607, using the image identifiers corresponding to the determined one ormore images, the image information overlay system determines imagelocations corresponding to the images identified as vehicle images bythe image classifier 105. The image locations may be locations in thedisplay of the hypertext document.

According to some embodiments, at step 608, the image informationoverlay system may display the overlay information on the one or moredetermined images, in the display of the hypertext document, based onthe feature locations and the image locations of the one or moredetermined images. The image information overlay system may display theoverlay information in response to the determined image being visible inthe display of the hypertext document and may update the content orposition of the displayed overlay information in response to changes inthe display of the hypertext document.

FIG. 7 is a non-limiting example diagram of a web browser 702 interfacefor image information overlay according to some embodiments.

According to some embodiments, a web browser 702 may be displayed in awindow of a graphical user interface 700 of a computer system. The webbrowser 702 may display the image 704 within the display of a hypertextmarkup language document. In this non-limiting example, the vehicle 705is displayed in a ¾ front view. The image classifier 105 may identifyone or more features of the vehicle 705 displayed in the image 704. As anon-limiting example, the hood 706 a of the vehicle 705 is identifiedand overlay information 706 b related to the vehicle 705 hood 706 a isdisplayed on or near the hood 706 a in the image 704. Other overlayinformation may also be displayed overlaid on a particular feature ofthe vehicle 705, related to the overlay information, or in a generallocation on the image 704.

FIG. 8 is a block diagram of an example computer system useful forimplementing various embodiments disclosed herein.

Various embodiments may be implemented, for example, using one or morewell-known computer systems, such as computer system 800 shown in FIG.8. One or more computer systems 800 may be used, for example, toimplement any of the embodiments discussed herein, as well ascombinations and sub-combinations thereof.

Computer system 800 may include one or more processors (also calledcentral processing units, or CPUs), such as a processor 804. Processor804 may be connected to a communication infrastructure or bus 806.

Computer system 800 may also include user input/output device(s) 803,such as monitors, keyboards, pointing devices, etc., which maycommunicate with communication infrastructure 806 through userinput/output interface(s) 802.

One or more of processors 804 may be a graphics processing unit (GPU).In an embodiment, a GPU may be a processor that is a specializedelectronic circuit designed to process mathematically intensiveapplications. The GPU may have a parallel structure that is efficientfor parallel processing of large blocks of data, such as mathematicallyintensive data common to computer graphics applications, images, videos,etc.

Computer system 800 may also include a main or primary memory 808, suchas random access memory (RAM). Main memory 808 may include one or morelevels of cache and/or registers. Main memory 808 may have storedtherein control logic (i.e., computer software) and/or data.

Computer system 800 may also include one or more secondary storagedevices or memory 810. Secondary memory 810 may include, for example, ahard disk drive 812 and/or a removable storage device or drive 814.Removable storage drive 814 may be a floppy disk drive, a magnetic tapedrive, a compact disk drive, an optical storage device, tape backupdevice, and/or any other storage device/drive.

Removable storage drive 814 may interact with a removable storage unit818. Removable storage unit 818 may include a computer usable orreadable storage device having stored thereon computer software (controllogic) and/or data. Removable storage unit 818 may be a floppy disk,magnetic tape, compact disk, DVD, optical storage disk, and/any othercomputer data storage device. Removable storage drive 814 may read fromand/or write to removable storage unit 818.

Secondary memory 810 may include other means, devices, components,instrumentalities or other approaches for allowing computer programsand/or other instructions and/or data to be accessed by computer system800. Such means, devices, components, instrumentalities or otherapproaches may include, for example, a removable storage unit 822 and aninterface 820. Examples of the removable storage unit 822 and theinterface 820 may include a program cartridge and cartridge interface(such as that found in video game devices), a removable memory chip(such as an EPROM or PROM) and associated socket, a memory stick and USBport, a memory card and associated memory card slot, and/or any otherremovable storage unit and associated interface.

Computer system 800 may further include a communication or networkinterface 824. Communication interface 824 may enable computer system800 to communicate and interact with any combination of externaldevices, external networks, external entities, etc. (individually andcollectively referenced by reference number 828). For example,communication interface 824 may allow computer system 800 to communicatewith external or remote devices 828 over communications path 826, whichmay be wired and/or wireless (or a combination thereof), and which mayinclude any combination of LANs, WANs, the Internet, etc. Control logicand/or data may be transmitted to and from computer system 800 viacommunication path 826.

Computer system 800 may also be any of a personal digital assistant(PDA), desktop workstation, laptop or notebook computer, netbook,tablet, smart phone, smart watch or other wearable, appliance, part ofthe Internet-of-Things, and/or embedded system, to name a fewnon-limiting examples, or any combination thereof.

Computer system 800 may be a client or server, accessing or hosting anyapplications and/or data through any delivery paradigm, including butnot limited to remote or distributed cloud computing solutions; local oron-premises software (“on-premise” cloud-based solutions); “as aservice” models (e.g., content as a service (CaaS), digital content as aservice (DCaaS), software as a service (SaaS), managed software as aservice (MSaaS), platform as a service (PaaS), desktop as a service(DaaS), framework as a service (FaaS), backend as a service (BaaS),mobile backend as a service (MBaaS), infrastructure as a service (IaaS),etc.); and/or a hybrid model including any combination of the foregoingexamples or other services or delivery paradigms.

Any applicable data structures, file formats, and schemas in computersystem 800 may be derived from standards including but not limited toJavaScript Object Notation (JSON), Extensible Markup Language (XML), YetAnother Markup Language (YAML), Extensible Hypertext Markup Language(XHTML), Wireless Markup Language (WML), MessagePack, XML User InterfaceLanguage (XUL), or any other functionally similar representations aloneor in combination. Alternatively, proprietary data structures, formatsor schemas may be used, either exclusively or in combination with knownor open standards.

In some embodiments, a tangible, non-transitory apparatus or article ofmanufacture comprising a tangible, non-transitory computer useable orreadable medium having control logic (software) stored thereon may alsobe referred to herein as a computer program product or program storagedevice. This includes, but is not limited to, computer system 800, mainmemory 808, secondary memory 810, and removable storage units 818 and822, as well as tangible articles of manufacture embodying anycombination of the foregoing. Such control logic, when executed by oneor more data processing devices (such as computer system 800), may causesuch data processing devices to operate as described herein.

Based on the teachings contained in this disclosure, it will be apparentto persons skilled in the relevant art(s) how to make and useembodiments of this disclosure using data processing devices, computersystems and/or computer architectures other than that shown in FIG. 8.In particular, embodiments can operate with software, hardware, and/oroperating system implementations other than those described herein.

It is to be appreciated that the Detailed Description section, and notthe Summary and

Abstract sections, is intended to be used to interpret the claims. TheSummary and Abstract sections may set forth one or more but not allexemplary embodiments of the present invention as contemplated by theinventor(s), and thus, are not intended to limit the present inventionand the appended claims in any way.

The present invention has been described above with the aid offunctional building blocks illustrating the implementation of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have been arbitrarily defined herein for the convenienceof the description. Alternate boundaries can be defined so long as thespecified functions and relationships thereof are appropriatelyperformed.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the invention that others can, by applyingknowledge within the skill of the art, readily modify and/or adapt forvarious applications such specific embodiments, without undueexperimentation, without departing from the general concept of thepresent invention. Therefore, such adaptations and modifications areintended to be within the meaning and range of equivalents of thedisclosed embodiments, based on the teaching and guidance presentedherein. It is to be understood that the phraseology or terminologyherein is for the purpose of description and not of limitation, suchthat the terminology or phraseology of the present specification is tobe interpreted by the skilled artisan in light of the teachings andguidance.

The breadth and scope of the present invention should not be limited byany of the above-described exemplary embodiments, but should be definedonly in accordance with the following claims and their equivalents.

The claims in the instant application are different than those of theparent application or other related applications. The Applicanttherefore rescinds any disclaimer of claim scope made in the parentapplication or any predecessor application in relation to the instantapplication. The Examiner is therefore advised that any such previousdisclaimer and the cited references that it was made to avoid, may needto be revisited. Further, the Examiner is also reminded that anydisclaimer made in the instant application should not be read into oragainst the parent application.

What is claimed is:
 1. An apparatus, comprising: a memory; and at leastone processor coupled to the memory, wherein the processor and thememory are configured to: send, to a server device, a selected imageidentifier for identification of at least one feature in an imagedisplayed in a hypertext document; receive, from the server device,identification data associated with the image, wherein theidentification data includes at least one feature identifier and atleast one feature location that corresponds to the at least one featureidentifier; retrieve overlay information based at least on a feature ofthe identification data; determine a location of the image displayed inthe hypertext document based at least on a layout of the display of thehypertext document; and display the overlay information on the imagebased at least on a feature location of the at least one featurelocation and the location of the image.
 2. The apparatus of claim 1,wherein the processor and the memory are further configured to: identifyan individual identifier in the hypertext document, wherein theindividual identifier includes a vehicle identification number (VIN)that is included in the hypertext document; and select the imageidentifier based at least on a proximity between the image identifierand the individual identifier in the display of the hypertext document.3. The apparatus of claim 1, wherein the processor and the memory arefurther configured to: display the overlay information at or near the atleast one feature identified in the image, wherein the overlayinformation includes at least one of: a maintenance issue information,engine specification information, mileage information, manufacturerrecall information, fuel-economy information, or any combinationthereof.
 4. The apparatus of claim 2, wherein the feature locationcorresponds to a feature identified in the image, the identificationdata further includes a general identifier, and the general identifierindicates a make and a model of a vehicle in the image.
 5. The apparatusof claim 4, wherein to retrieve the overlay information, the memory andthe at least one processor is configured to retrieve the overlayinformation from at least one vehicle database, wherein the overlayinformation corresponds to the general identifier.
 6. The apparatus ofclaim 1, wherein the server device is configured to: classify the imageidentified by the image identifier to identify the image in at least oneimage class; and identify the at least one feature and the at least onefeature location that corresponds to the at least one feature.
 7. Theapparatus of claim 6, wherein the server device is further configuredto: generate the identification data, wherein the identification dataincludes the at least one feature location, the at least one featureidentifier corresponding to the at least one feature, and the imageidentifier corresponding to the image.
 8. A computer-implemented method,comprising: sending, to a server device, a selected image identifier foridentification of at least one feature in an image displayed in ahypertext document; receiving, from the server device, identificationdata associated with the image, wherein the identification data includesat least one feature identifier and at least one feature location thatcorresponds to the at least one feature identifier; retrieving overlayinformation based at least on a feature of the identification data;determining a location of the image displayed in the hypertext documentbased at least on a layout of the display of the hypertext document; anddisplaying the overlay information on the image based at least on afeature location of the at least one feature location and the locationof the image.
 9. The computer-implemented method of claim 8, furthercomprising: identifying an individual identifier in the hypertextdocument, wherein the individual identifier includes a vehicleidentification number (VIN) that is included in the hypertext document;and selecting the image identifier based at least on a proximity betweenthe image identifier and the individual identifier in the display of thehypertext document.
 10. The computer-implemented method of claim 8,wherein the overlay information is displayed at or near the at least onefeature identified in the image, and the overlay information includes atleast one of: a maintenance issue information, engine specificationinformation, mileage information, manufacturer recall information,fuel-economy information, or any combination thereof.
 11. Thecomputer-implemented method of claim 9, wherein the feature locationcorresponds to a feature identified in the image, the identificationdata further includes a general identifier, and the general identifierindicates a make and a model of a vehicle in the image.
 12. Thecomputer-implemented method of claim 11, wherein the retrieving of theoverlay information further comprises retrieving the overlay informationfrom at least one vehicle database, wherein the overlay informationcorresponds to the general identifier.
 13. An apparatus, comprising: amemory; and at least one processor operatively coupled to the memory,wherein the processor and the memory are configured to: retrieve animage identified by a corresponding image identifier for identificationof at least one feature; classify the image using a first machinelearning model in a first general image class; classify the image usinga second machine learning model to generate a general identifier for theimage, the general identifier corresponding to a sub-class of aplurality of sub-classes within the first general image class; identifya feature and a feature location corresponding to the feature within theimage; and generate identification data comprising a feature identifiercorresponding to the feature, the feature location corresponding to thefeature, and the image identifier corresponding to the image.
 14. Theapparatus of claim 13, wherein the processor and the memory are furtherconfigured to: receive, from a client device, the image identifier foridentification of at least one feature in the image, and send, to theclient device, the identification data, in response to receiving theimage identifier.
 15. The apparatus of claim 13, wherein the processorand the memory are configured to identify the feature and the featurelocation within the image using a third machine learning model.
 16. Theapparatus of claim 13, wherein the image is an exterior image or aninterior image of a vehicle having an associated vehicle identificationnumber (VIN), and the feature identified within the image includes atleast one of: body style, trim level, body kit, spoilers, scoops,accessory rack, number of doors, wheel size, alloy wheels, or anycombination thereof.
 17. The apparatus of claim 15, wherein the imageidentifier includes a universal resource locator (URL) identifying alocal or remote storage location that stores the image, and the featureidentifier represents a feature of a vehicle that is identified in theimage.
 18. The apparatus of claim 15, wherein the general identifiercomprises at least one of: a vehicle make, a vehicle model, or anycombination thereof; and the third machine learning model is configuredto identify the feature based at least on an image set corresponding tothe general identifier.
 19. The apparatus of claim 18, wherein the firstgeneral image class includes a plurality of vehicles images, and theplurality of vehicle images includes at least one vehicle interiorimage, at least one vehicle exterior image, or any combination thereof.20. The apparatus of claim 19, wherein the third machine learning modelis configured to identify at least one vehicle exterior feature in theat least one vehicle exterior image or at least one vehicle interiorfeature within the vehicle interior image.